Shannon entropy: a rigorous mathematical notion at the crossroads between probability, information theory, dynamical systems and statistical physics
نویسنده
چکیده
Statistical entropy was introduced by Shannon as a basic concept in information theory, measuring the average missing information on a random source. Extended into an entropy rate, it gives bounds in coding and compression theorems. I here present how statistical entropy and entropy rate relate to other notions of entropy, relevant either to probability theory (entropy of a discrete probability distribution measuring its unevenness), computer sciences (algorithmic complexity), the ergodic theory of dynamical systems (Kolmogorov-Sinai or metric entropy), or statistical physics (Boltzmann entropy). Their mathematical foundations and correlates (entropy concentration, Sanov, Shannon-McMillan-Breiman, Lempel-Ziv and Pesin theorems) clarify their interpretation and offer a rigorous basis to maximum entropy principles. Although often ignored, these mathematical perspectives give a central position to entropy and relative entropy in statistical laws describing generic collective behaviors. They provide insights into the notions of randomness, typicality and disorder. The relevance of entropy outside the realm of physics, for living systems and ecosystems, is yet to be demonstrated.
منابع مشابه
Shannon entropy: a rigorous notion at the crossroads between probability, information theory, dynamical systems and statistical physics
متن کامل
Entropy of infinite systems and transformations
The Kolmogorov-Sinai entropy is a far reaching dynamical generalization of Shannon entropy of information systems. This entropy works perfectly for probability measure preserving (p.m.p.) transformations. However, it is not useful when there is no finite invariant measure. There are certain successful extensions of the notion of entropy to infinite measure spaces, or transformations with ...
متن کاملSome properties of the parametric relative operator entropy
The notion of entropy was introduced by Clausius in 1850, and some of the main steps towards the consolidation of the concept were taken by Boltzmann and Gibbs. Since then several extensions and reformulations have been developed in various disciplines with motivations and applications in different subjects, such as statistical mechanics, information theory, and dynamical systems. Fujii and Kam...
متن کاملDynamics of Uncertainty in Nonequilibrium Random Motion
Shannon information entropy is a natural measure of probability (de)localization and thus (un)predictability in various procedures of data analysis for model systems. We pay particular attention to links between the Shannon entropy and the related Fisher information notion, which jointly account for the shape and extension of continuous probability distributions. Classical, dynamical and random...
متن کاملCombinatorial entropies and statistics
We examine the combinatorial or probabilistic definition (“Boltzmann’s principle”) of the entropy or cross-entropy function H ∝ lnW or D ∝ − lnP, where W is the statistical weight and P the probability of a given realization of a system. Extremisation of H or D, subject to any constraints, thus selects the “most probable” (MaxProb) realization. If the system is multinomial, D converges asymptot...
متن کامل